What does the representation talk back to you?

Our research goal is to support designers with interactive systems by exploring the relationship among representations, their meanings, and their effects. This paper first outlines ART (Amplifying Representational Talkback), a prototype system which we have developed to instantiate our research framework. We then reinterpret the essence of the ART system by illustrating the ART approach with other people’s work in Semiotic Approaches to User Interface Design presented at the ACM CHI 2000 workshop. We identify critical aspects of the system from four points: (1) interactions with representations “I” produced; (2) representations as indices for thoughts; (3) hands-on representations; and (4) limiting the automation. By having the ART system as an object-to-thing-with, we argue that communication with interactive computational tools had better been regarded as interaction with representations.